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(54) Method for controllably deinterlacing sequential lines of video data fields based upon pixel 
signals associated with four successive interlaced video fields 



(57) A hybrid deinterlace mechanism employs 
motion detection as a precursor to select the value for 
each missing pixel of adeinterlaced frame, which is gen- 
erated by using four consecutive fields to produce a 
motion detection map that is subsequently coupled to a 
deinterlacer. together with a selected two of the four con- 
secutive fields. These two selected fields and the motion 
map are sufficient for the deinterlacer to completely char- 
acterize the resulting deinterlaced frame. The deinter- 
lacer switches between vertical interpolation and 



merging based upon motion. Motion is detected by 
applying a logic OR operation to respective even and odd 
field binary motion detection signals. In the absence of 
motion, the pixel values of an odd field and its immedi- 
ately succeeding even field are merged. On the other 
hand, if motion is detected, vertical interpolation is car- 
ried out for the even field, by a linear combination of 
immediately adjacent pixels, respectively above and 
below the pixel of interest. 
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Description 

FIELD OF THE INVENTION 

s The present invention relates in general to an image signal processing technique for converting interlaced video 

signals into progressive or sequential signal format, and is particularly directed to a multiple interlaced image field 
processing mechanism, that is operative to generate pixel signals within each of sequential lines of a deinterlaced video 
data field associated with a prescribed field polarity, by selectively merging pixels from an immediately adjacent image 
field or performing interpolation among pixels of the field of interest, in dependence upon a determination of the presence 

10 or absence of image motion between a first pair of image fields of a first polarity and a second pair of image fields of an 
opposite polarity, wherein one of the fields of the two pairs is the image field of interest. 

BACKGROUND OF THE INVENTION 

is In order to display interlaced video signals on a progressive or sequential line display with high visual quality, it is 
necessary to 'deinterlace' interlaced odd and even fields of video signals into a sequentially continuous field of pixels 
without (interline) gaps. In addition, deinterlacing, prior to printing, results in higher quality still pictures from video. 

In an interlaced video frame, each frame of pixels is composed of two time sequential fields - an even field and an 
odd field. Compared to a full frame of successive lines without missing lines of pixels, each (odd or even) field is sub- 

20 sampled by a factor of two in the vertical dimension, so that, as diagrammatically illustrated in Figure 1 , an even field 
contains data at only even-numbered line locations (e.g. lines 0, 2, 4, 6, and 8), and an odd field contains data at only 
odd-numbered line locations (e.g. lines 1, 3, 5, 7,). Thus, an even field has no pixel values for odd numbered lines of 
the full frame, and an odd field has no pixel values for even numbered line of the full frame. 

In order to deinterlace an even (or an odd) field into a full frame without missing lines of pixels, it is necessary to 

25 estimate the missing odd (or even) lines. One well-known method for this purpose involves merging the even and odd 
fields together, namely simply filling in the missing lines of the odd (even) field with the lines of the immediately adjacent 
even (odd) field. Unfortunately, such an approach introduces "jitter" artifacts at portions of the image containing moving 
objects (i.e. objects that move within the time interval of two successive fields). On the other hand, merging provides 
optimal spatial resolution at steady image regions (namely at those pixel locations where the image does not change 

30 between successive fields) . 

Another approach is to concentrate on a single field only (i.e., the odd field) and interpolate the missing lines using 
spatial interpolation. One example of a relatively simple spatial interpolation technique involves bilinear interpolation, in 
which an average of the available pixel values in lines immediately above and below the pixel of interest in the missing 
line is assigned to the missing pixel. However, this method may cause artifacts if the missing pixel is over an edge whose 

35 orientation is not vertical. 

To overcome these artifacts, an edge-adaptive spatial interpolation method, described in the U.S. Patent to Dougall 
et al., No. 5,019,903, entitled "Spatial Interpolation Between Lines Of A Supersampled Digital Video Signal In Accordance 
With A Gradient Vector Selected For Maximum Matching Of Blocks Of Samples Which Are Offset In Opposite Directions," 
has been proposed. The patented technique first attempts to determine the orientation of the image gradient at the 

40 missing pixel, and then interpolation is performed using image values that are located along this determined orientation, 
in order not to "cross an edge" and cause unwanted artifacts. 

The Dougall et al. patent proposes that a potentially more effective method would be to use a hybrid scheme, where 
the deinterlacing process would switch, on a pixel-by-pixel basis, between merging and (edge-adaptive) spatial interpo- 
lation, depending on the dynamics of the image at the locations of the missing pixels, so that the reduced complexity 

45 advantages of using merging in steady regions of the image would be maintained. 

In order to classify the dynamics of each pixel as either a "moving pixel" or "steady pixel," it would be necessary to 
employ a motion detection scheme as a precursor to choosing merging or interpolation. However, the Dougall et al 
patent offers no discussion as to how to implement such a mechanism. 

In order to detect motion in an image, the contents of successive image fields of opposite polarity (even-odd or odd- 

so even) can be compared with one another. However, the accuracy of motion detection can be increased significantly 
when two consecutive fields of the same polarity (i.e., an immediately preceding even (odd) field (i-1) and an immediately 
succeeding even (odd) field (i+1)), between which an odd (even) field (i) occurs, are utilized for motion detection. 

The U.S. patent to Bennett et al., No. 4,472,732 describes a method which employs the pixel-to-pixel difference of 
neighboring fields with the same polarity (e.g. even fields) that immediately follow and precede the field to be deinterlaced 

55 (e.g. an odd field), in order to perform motion detection. The method then switches between merging and vertical inter- 
polation depending upon the presence and absence of motion that is determined by thresholding the difference values. 

Unfortunately, the use of single-pixel differences may falsely detect 'no motion* if the scene is such that the gray 
levels of the pixels being compared in the two neighboring fields are similar, even though there is motion in the scene. 
Such a situation may occur, for instance, in the case of scenes that contain a moving structure with text on it. 
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An example of this circumstance is illustrated in Figure 2, which diagrammatically depicts what is essentially a black 
letter 'E' rotated clockwise by 90° and situated against a white background. For this illustrated image, a *no motion' 
decision is rendered at the (missing) pixel location (x,y) in the field at time t2, by differencing co-located values at fields 
at times t1 and t3, although there is motion at that pixel location. In this case, merging the fields at times t1 and t2 at 

5 pixel location (x,y) will result in artifacts due to this false classification. 

The occurrence of such false classifications may be reduced by differencing and thresholding an N1xN2 block of 
pixels (e.g. N1=N2=3) rather than a single pixel. The benefits of using a 1x2 block (i.e. N1=1, N2=2), for instance, and 
thresholding the difference (1/2)(|f(x,y,t1) - f(x,y,t3)| + |f(x+1,y,t1) - f(x+1 p y,t3)|) to detect whether or not the missing pixel 
f(x,y,t2) belongs to a moving region may be seen in Figure 3, where f(.,.,.) denotes the spatiotemporal distribution of the 

io fields. 

If, as shown at time t4 in Figure 4, a further fourth field is used for motion detection, a comparison of the fields at 
times t2 and t4 (in addition to the comparison of the fields at times t1 and t3) may increase the reliability of motion 
detection. In the example of Figure 4, a "moving" decision can be rendered for the pixel of interest in the frame at time 
t2 as a result of comparing the corresponding pixels (at spatial location (x,y+1)) at frames at times t2 and t4. 
15 The U.S. patent to Ishikawa, No. 4,785,351 describes a deinterlacing scheme, as applied to a time sequence of 
interlaced f ields, such as an interlaced TV video signal, where motion detection is based not only upon the pixel-to-pixel 
difference of immediate neighbors (e.g. fields at t2 and t4, shown in Figure 5), but also on the basis of a motion detection 
signal that has been obtained previously for deinterlacing t2, using t1 and t3, so as to effectively employ four consecutive 
fields. 

20 In the scheme detailed in the '351 patent, in an effort to reduce noise, an average of existing pixel values of the 
fields surrounding the field of interest is placed in the location of the missing pixel, rather than directly using the value 
of a neighboring pixel (e.g. merging). However, since the existing lines in the field of interest that are directly reproduced 
in the deinterlaced version of the image are not processed for noise reduction, the averaging process for missing pixels 
may introduce some undesirable artifacts in the resulting image frame. Although such artifacts may not be visible in TV 

25 viewing applications, where the resulting frames are displayed in motion, they may be objectionable when the resulting 
deinterlaced frames are reproduced as still images (lor display or printing). For example, over a stationary region, the 
missing lines that are generated by temporal interpolation may appear less noisy than the adjacent unprocessed (exist- 
ing) field lines, causing a visually disturbing effect, especially noticeable in printing applications. 

The effect of the deinterlacing scheme described in the '35 1 patent is illustrated in Figure 5. The patented mechanism 

30 employs a rolling memory system that is a cascade of two field stores and a line buffer, and makes the resulting dein- 
terlaced frame available, on a pixel-by-pixel basis, to a display system, for example, in the case of a television platform 
performing interlaced to progressive scan conversion. In Figure 5, as the field data (E2) at time t3 becomes available, 
motion detection is performed by taking the difference between values (E1) and (E2) from fields at times t1 and t3, 
respectively, e.g. the absolute difference |a3 - c3|, and comparing the difference with a threshold in order to detect motion 

35 for missing pixel value denoted as X1 . 

The binary motion detection decision is reflected by appropriately setting the value of a motion detection signal me 
(e.g. me = 1 if there is motion). Depending upon the value of me, vertical interpolation (e.g. X1 = (1/2)(b2+b3) ) or 
temporal interpolation (e.g. X1 = (1/2)(a3+c3) ) is performed to produce the values of the deinterlaced frame at time t2. 
Next, when the field at time t3 is deinterlaced, the fields at times t2 and t4 are employed for motion detection, e.g., 

40 absolute difference |b3-d3| is compared with a threshold to produce a binary motion detection signal mo. The motion 
detection signal for the missing pixel X2 is defined by logically ORing me and mo, i.e.. md = (me)V(mo) . In other words, 
the previous motion detection signal is stored and used for the next frame. It should be noted that the system described 
in the '351 patent produces deinterlaced frames in a sequential manner. For example, in the case shown in Figure 5, 
the Ishikawa system first produces a deinterlaced version of the odd field 01 and then a deinterlaced version of the even 

45 field E2. 

Another four-field based motion detection approach is described in the U.S. patent to Motoe et at, No. 5,021 ,870, 
where the missing pixel of interest is set to a value that is equal to a linear combination of the result of temporal averaging 
of values belonging to the surrounding fields that have the same parity, and the result of vertical spatial interpolation 
performed using the existing pixel values of the field of interest. The motion detection signal determines the weights 
so assigned to these two quantities in forming the linear combination. This method uses a rolling memory system formed 
of three pixel stores and supplies the values of the deinterlaced frame pixel by pixel to a display system. As in the above- 
referenced '351 patent, pixel-by-pixel differences are employed to generate a motion detection signal. 

SUMMARY OF THE INVENTION 

55 

In accordance with the present invention, a single deinterlaced frame is generated by using four consecutive fields 
to produce a motion detection map that is subsequently coupled to a deinterlacer, together with a selected two of the 
four consecutive fields. These two selected fields and the motion map are sufficient for the deinterlacer to completely 
characterize the resulting deinterlaced frame. 
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In a preferred embodiment of the invention, rather than employ temporal averaging employed in the above-referenced 
'870 patent to Motoe et al, the deinterlacer switches between vertical interpolation and merging based upon motion. 
Motion is detected by applying a logic OR operation to respective even and odd field binary motion detection signals. 
In the absence of motion, the pixel values of an odd field and its immediately succeeding even field are merged. On the 

5 other hand, if motion is detected, vertical interpolation is carried out for the even field, by a linear combination of imme- 
diately adjacent pixels, respectively above and below the pixel of interest. 

Merging or spatial interpolation is performed to estimate the values of the missing pixels, with one of the factors 
being the presence or absence of motion in the image at the locations of the missing pixels. Motion detection is performed 
for each one of the missing pixels, and the result is stored as a binary value in the respective storage locations of a 

10 motion detection map storage space. In generating the motion detection map a first binary value (e.g. a "1") is written 
to respective one of the storage locations of the motion detection storage space, if image motion is detected for the 
missing pixel location xi of interest. Conversely, a second binary value (e.g. a "O") is stored if there is no image motion 
for that missing pixel location. 

The system of the invention employs odd and even field stores, which respectively store the contents of selected 

is odd and even fields. The contents of the motion detection map store and the field stores are coupled to a deinterlacer. 
The deinterlacer produces a respective missing pixel. In a preferred embodiment of the invention, vertical bilinear inter- 
polation is employed, using the average of the existing pixel values located immediately above and below the missing 
pixel. 

Motion detection is performed in two steps. In the first step, the pixels of the same respective locations of the two 
20 even fields are compared with one another to form a first 'even* binary motion detection array. Likewise, in a second 
step, the pixels of the same respective locations of the two odd fields are compared with one another to form a second 
'odd* binary motion detection array. A first even motion detection array is computed by comparing a sum signal, repre- 
sentative of the average of differences between values of respective ones of pixels of arrays of pixels which neighbor 
the corresponding pixel location in each of the first and second image fields, with a threshold T. If the sum signal exceeds 
25 the threshold value T, then it is concluded that there is effective relative image motion between the first and second 
image fields, and a first binary value (e.g. a "1 ") is written into storage location of the even field storage space. Otherwise, 
a second binary value (e.g. a "O") is written into the storage location, to indicate that there is no relative image motion 
at the location of the missing pixel of the even field. 

Where the location of the missing pixel of a selected even field is at a border location of the image field, the size of 
30 the array of pixels employed to perform motion detection and pixel interpolation are reduced. For the first and last row, 
and the first and last column, the three-by-three comparison window may be downsized, for example to a two-by-two 
window and to a two-by-three window, respectively. 

In addition to computing a motion detection map for an even field, a motion detection map is computed for a selected 
odd field, again on the basis of thresholding a block of (NxN) pixels. Once the two intermediate motion detection arrays 
35 for the odd and even field pairs have been derived, a final motion detection map is generated by logically ORing, point- 
by-point, the contents of the two motion detection arrays. The deinterlacer then proceeds to process the odd and even 
fields supplied by the upstream motion detection system. 

When processing the successive image fields of the four field sequence, the respective pixel data values of a first 
even field are written into storage locations of an even field store, and the respective pixel data values of a first odd field 
40 are written into storage locations of an odd field store. For the third (even) field in the four field sequence, respective 
pixel values are sequentially written into an N-line auxiliary buffer. 

In the course of loading the third field pixels into the auxiliary buffer, the buffer will eventually contain a sufficient 
number of pixel values to permit a binary motion value to be computed from its contents and those of the image store 
for the even field. When the buffer becomes full, a binary motion detection value is computed for a missing pixel location, 
45 and stored in the motion detection memory. This process is sequentially executed for successive pixels, computing 
successive binary values of the motion detection memory. 

Once the last pixel value is loaded into the last line of the auxiliary buffer, further loading of the buffer cannot proceed 
unless accommodation is made for replacing already loaded buffer space with additional pixels of the image field. In 
accordance with the invention, respective lines of pixel values of the buffer are successively transferred into successive 
so lines of the even field memory and written over the current pixel values of the replaced lines. As new pixel values are 
written into the buffer, binary motion detection values are computed and written into the motion detection memory. 

Eventually, through successive repetitions of the above transfer and replacement operations between the auxiliary 
buffer and the even field memory, the even field memory will contain all the pixels of the field that have been transferred 
from the auxiliary buffer B, the odd field memory contains all the pixels of the odd field, and the motion detection memory 
55 contains motion array binary values for the even fields. 

After all motion detection values for the even fields have been computed the process initiates a similar process for 
the odd field and, in addition, uses the already computed even field data to logically combine, on the fly, the motion 
detection values between the even and odd fields. Upon completion of the odd field routine, the even field memory 
contains all the pixels of the selected even field to be supplied to the deinterlacer, the odd field memory contains all the 
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pixels of the selected odd field to be supplied to the deinterlacer, and the motion detection memory contains the logical 
sums of the even and odd motion array binary values as final motion detection values. As a result, the deinterlacer has 
all the necessary information to generate a deinterlaced frame. 

5 BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 diagram matically illustrates an interlaced video frame composed of two time sequential fields - an even 
field and an odd field; 

Figure 2 depicts motion detection for a missing pixel located at spatial location (x,y) within a frame at time t2 using 
10 the existing co-located pixels within neighboring fields; 

Figure 3 illustrates increased reliability in motion detection by comparing a one-by-two block of pixels; 

Figure 4 illustrates the use of a fourth field for increasing the reliability of motion detection; 

Figure 5 diagrammatically illustrates the effect of the deinterlacing scheme described in the U.S. patent to Ishikawa, 

No. 4,785,351 ; 

15 Figure 6 diagram matically illustrates a four field-based deinterlacing system in accordance with the present inven- 
tion; 

Figure 7 diagrammatically illustrates the manner in which a deinterlacer of a preferred embodiment of the present 
invention switches between vertical interpolation and merging; 

Figure 8 diagrammatically illustrates a non-limiting example of four successive image fields E1-01-E2-O2 of inter- 
20 laced video-representative pixels that are processed to produce a motion detection map; 
Figure 9 diagrammatically illustrates a map storage space; 

Figures 10(a) - 10(c) diagrammatically show the manner in which successive lines of an even field can be stored in 
an N-line buffer and used to generate the values of the motion detection map stored in a motion decision storage 
space; and 

25 Figures 11 (a) - 11 (c) diagrammatically show the manner in which successive lines of an odd field can be stored in 
an N-line buffer and used to generate the values of the motion detection map stored in a motion decision storage 
space. 

DETAILED DESCRIPTION 

30 

Before describing in detail the new and improved deinterlacing mechanism in accordance with the present invention, 
it should be observed that the present invention resides primarily in what is effectively a prescribed digital image process- 
ing technique that may be implemented by means of conventional digital signal processing circuitry, in particular, delay 
(shift register) components and arithmetic logic circuits, or may be embedded within image processing application soft- 

35 ware executable by the control processor of a digital image processing workstation, through which successive interlaced 
image fields of a scene are processed. The sources of such interlaced image fields may be conventional imaging hard- 
ware, such as interlaced lines of television signals derived from an NTSC system, as a non-limiting example, and are 
not considered part of the invention. 

Consequently, the manner in which such interlaced image field sources are interfaced with either hardwired digital 

40 circuit components or a digital image processing workstation have been illustrated in the drawings in readily understand- 
able block diagram format, which show only those specific details that are pertinent to the present invention, so as not 
to obscure the disclosure with details which will be readily apparent to those skilled in the art having the benefit of the 
description herein. Thus, the block diagram illustrations are primarily intended to illustrate the major components of the 
system in a convenient functional grouping, whereby the present invention may be more readily understood. 

45 Figure 6 diagrammatically illustrates a four field-based deinterlacing system in accordance with the present invention 
as comprising a front end component comprising a memory efficient four field motion detection system 61 which is 
coupled to receive as inputs, four successive even and odd image fields E 1 , 01 , E2 and 02, and is operative to selectively 
output two of its four input fields (e.g. fields 01 and E2) and a motion detection map (md) to a downstream deinterlacer 
unit 62, from which the deinterlaced frame is derived. 

so As diagrammatically illustrated in Figure 7, in accordance with a preferred embodiment of the present invention, the 
deinterlacer switches between vertical interpolation and merging, not using temporal averaging employed in the above- 
referenced '870 patent to Motoe et, in which the missing pixel is set to a linear combination of the result of temporal 
averaging of values belonging to the surrounding fields having the same polarity, and the result of vertical spatial inter- 
polation performed using the existing pixel values of the field of interest. 

55 In particular, motion is detected by applying a logic OR operation to the two binary motion detection signals me and 
mo. In the absence of motion, the pixel values of the first odd field (Ol) and its immediately succeeding (second) even 
field (E2) are merged. Otherwise (where motion is detected), vertical interpolation is carried out for field E2, as shown 
by the linear combination of immediately adjacent pixels c3 and c4, respectively above and below the pixel X of interest. 
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Generating motion detection signals using the differences computed on the basis of a block of signals will be described 
bei:w with reference to Figures 8 - 1 1(c). 

fgure 8 diagrammatically illustrates a non-limiting example of four successive image fields E1 -01 -E2-02 of inter- 
laced video- representative pixels that are processed to produce a motion detection map which, together with a selected 

5 two of the four consecutivef ields, are employed to completely characterize the resulting deinterlaced frame. For purposes 
of simplification, in the present example, each of the even and odd fields is diagrammatically illustrated as an array of 
M1 lines by M2 pixels per line (e.g. five lines of six pixels per line, as shown), with the odd fields (01) and (02) being 
offset by one line from the even fields (E1) and (E2). 

For the first field (even field E1) of the four field sequence, the available field pixels are denoted by a1 , a2,..., a30, 

10 and for the next successive even field E2, the available pixels are denoted by c1 , c2 c30. For the second image field 

(odd field 01) of the four field sequence, which is immediately adjacent and intermediate the even fields E1 and E2, the 
available pixels are denoted by bl , b2,..., b30. For the fourth field (odd field 02) of the four field sequence, which imme- 
diately succeeds even field E2, the available pixels are denoted by d1 , d2 d30. 

In the present example, the missing pixels of the missing lines of even field E2 are denoted by x1 , x2,..., x30. The 

15 problem of deinterlacing is to estimate the missing pixel values x1, x2,..., x30 of even field E2 using the available pixel 
values of each of the four fields: E1-01-E2-02. 

Pursuant to the invention, merging or spatial interpolation is performed to estimate the values of the missing pixels 
of the third field in the sequence (even field E2), with one of the factors being the presence or absence of motion in the 
image at the locations of the missing pixels. Motion detection is performed for each one of the missing pixels x1 , x2 

20 x30, and the result is stored as a binary value in the respective storage locations ml , m2 m30 of the M1xM2 motion 

detection map storage space diagrammatically illustrated at M in Figure 9. 

As will be described, a respective one of the storage locations ml - m30 of the motion detection storage space M 
of Figure 9 will contain either a "1" (if image motion is detected for the missing pixel location xi of interest), or a "O" (if 
there is no image motion for that missing pixel location). 

25 Also shown in Figure 9 are respective field stores FS1 and FS2, to which the contents of respective image fields 
are written as field values q1 -q30 and r1 -r30. Also shown in Figure 9 is an N-line buffer B (where N«M1 , M2). For the 
five line (Ml), by six pixels per line (M2) example herein described, N is set to three, having storage locations si - s1 8. 
As described above, the contents of motion detection map store M and field stores FS1 and FS2 provided by the four 
field motion detection system 61 are coupled to deinterlacer 62 of Figure 6. Before describing how motion detection 

30 system 61 processes the four consecutive fields to produce the appropriate information for storage in stores FS1 , FS2 
and M, the operation of the deinterlacer 62 will be described. 

The operation of the deinterlacer 62 to produce a respective missing pixel xi is describable by the following relation- 
ship: 

35 xi = bi, if mi = 0 (1) 

y, otherwise, 

where y is a function of neighboring existing pixel values, in general, implementing a spatial interpolation, such 

40 as directional interpolation or bilinear vertical interpolation. As will be described, in accordance with an embodiment of 
the invention, vertical bilinear interpolation is employed, with y being equal to the average of the existing pixel values 
located immediately above and below the missing pixel, e.g., for a missing pixel at location x9 in the even field (E2) of 
Figure 8. i = 9, and y = (1/2)(c9+c15) . 

Motion detection is performed in two steps. In the first step, the pixels of the same respective locations of the even 

45 fields E1 and E2 are compared with one another to form a first 'even* binary motion detection array Me, which, for the 
present example of a five line by six pixels per line field size, is comprised of binary values me1 - me30, where mei = 1 
in the case of motion and mei = 0 for the no motion case. Similarly, in a second step, the pixels of the same respective 
locations of the odd fields 01 and 02 are compared with one another to form a second 'odd' binary motion detection 
array Mo, which, for the present example, is comprised of binary values mo1 - mo30, where moi = 1 in the case of motion 

so and moi = 0 for the no motion case. 

The first even motion detection array Me is computed by comparing the difference between N x N blocks of pixels 
from fields E1 and E2 with a threshold. As a non-limiting example, an N x N block may correspond to a 3 x 3 block of 
pixels. For instance, for the missing pixel location x9 within a three-by-three array E2A in the second even field E2, there 
is an associated array ElAfor immediately preceding even field E1. 

55 For a corresponding pixel location in each of thefirst (El) and second (E2) image fields, a sum signal representative 
of the average of differences between values of respective ones of pixels of the arrays El A and E2A of pixels neighboring 
the corresponding pixel location in each of the first and second image fields is generated. In the present example of a 
three-by-three summation array, this sum signal value Sum may be represented for missing pixel location x9 as: 



6 



BNSDOCID: <EP 0690617A2J_> 



EP 0 690 617 A2 



Sum x9 = (1/9)*(|a2-c2| + |a3-c3| + |a4-c4| + |a8-c8| + |a9-c9| + |a10-c10| + |a14-c14|+ (2) 

s |a15-c15| + |a16-c16|). 

This sum signal Sum value is compared with a threshold T. If the sum signal exceeds the threshold value T, then it 
is concluded that there is effective relative image motion between the first and second image fields, and a "1" is written 
into storage location (m9=me9 ) of the even field storage space represented by the array M in Figure 9. Otherwise, a 
10 "O" is written into storage location (m9=me9 ), indicating that there is no relative image motion at the location of missing 
pixel x9 of the even field E2. 

Namely, me9 = 1 if Sum x9 > T (3) 
15 0, otherwise. 

Similarly, considering missing pixel location x21, its associated sum signal value Sum^i may be represented as: 
Sum x21 = (1 /9)*(|al4 - c14| + |a15 - c15| + |a16 - c16| + |a20 - c20| + |a21 - c21| + |a22 - c22| + (4) 

20 

|a26 - c26| + |a27 - c27| + |a28 - C28| ). 

Again, the sum signal Sum 2 i value is compared with threshold T. If the sum signal exceeds the threshold value T, 
then it is concluded that there is effective relative image motion between the first and second even image fields at this 
25 pixel location, and a "1" is written into storage location (me=m21 ) of the storage space represented by the array M in 
Figure 9. Otherwise, a "O" is written into storage location (me=m21 ), indicating that there is no relative image motion 
at the location of missing pixel x21 . 

Namely, me21 =1 , if Sum x21 > T (5) 

30 

0, otherwise. 

Where the location of the missing pixel of the second even field E2 is at a border location of the image field, the size 
of the array of pixels employed to perform motion detection and pixel interpolation are reduced. For the first and last 
35 row, and the first and last column, the three-by-three comparison window is appropriately downsized to a two-by-two 
window and to a two-by-three window, respectively. 

More particularly, with reference again to Figure 8, for missing pixel location x1 adjacent to the upper left hand border 
corner of the second even field image frame E2 of Figure 8, the sum signal value Sum x1 may be derived from a two-by- 
two pixel array located at the upper left hand corner of each of the even fields E1 and E2. Namely, the sum signal value 
40 associated with missing pixel x1 may be expressed as: 

Sum x1 = (1/4)*(|a1-c1| + |a2-c2| + |a7-c7| + |a8-c8|). (6) 

If the sum signal value Sum x1 exceeds the threshold value T, then it is concluded that there is effective relative 
45 image motion between the first and second image fields at missing pixel location x1, and a "1" is written into storage 
location me1 of the storage space associated with the even field E2 represented by the array M in Figure 9. Otherwise, 
a "O" is written into storage location me1 , indicating that there is no relative image motion at the location of missing pixel 
x1. 

Similarly, for missing pixel location x2 adjacent to missing pixel x2, the sum signal value Sum^ may be derived from 
so a three-by-two pixel array located at the upper left hand corner of each of the even fields E1 and E2. Namely, the sum 
signal value associated with missing pixel x2 may be expressed as: 

Sum^ = (1/6)*(|a1-c1 1 + |a2-c2| + |a3-c3| + |a7-c7| + |a8-c8|) + |a9-c9|). (7) 

55 Again, if the sum signal value Sum^ exceeds the threshold value T, then it is concluded that there is effective relative 
image motion between the first and second image fields at missing pixel location x2, and a "1" is written into storage 
location me2 of the storage space associated with the even field E2 represented by the array M in Figure 9. Otherwise, 
a "O" is written into storage location me2, indicating that there is no relative image motion at the location of missing pixel 
x2. 
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In a like manner, considering missing pixels x26 and x30 at the lower portion of the even image field E2, for missing 
pixel location x26 adjacent to the lower left hand corner, the sum signal value Sum x26 may be derived from an associated 
three-by-two pixel array located at the lower left hand corner of each of the even fields E1 and E2. Namely, the sum 
signal value associated with missing pixel x26 may be expressed as: 

5 

Sum x26 = (1/6)*(|a19-c19| + |a20-c20| + |a21-c21| + |a25-c25| + |a26-c26|) + |a27-c27|). (8) 

If the sum signal value Sum^e exceeds the threshold value T, then it is concluded that there is effective relative 
image motion between the first and second image fields at missing pixel location x26, and a "1 " is written into storage 
10 location me26 of the storage space associated with the even field E2 represented by the array M in Figure 9. Otherwise, 
a "O" is written into storage location me26, indicating that there is no relative image motion at the location of missing 
pixel x26. 

For missing pixel location x30 at the lower right hand border corner of the second even field image frame E2 of 
Figure 8, its associated sum signal value Sum^o may be derived from a two-by-two pixel array located at the lower right 
15 hand corner of each of the even fields E1 and E2. Thus the sum signal value Sum x30 may be expressed as: 

Sum x30 = (1/4)*(|a23-c23| + |a24-c24| + |a29-c29| + |a30-c30|). (9) 

If the sum signal value Sum x30 exceeds the threshold value T, then it is concluded that there is effective relative 
20 image motion between the first and second image fields at missing pixel location x30, and a "1" is written into storage 
location me30 of the storage space associated with the even field E2 represented by the array M in Figure 9. Otherwise, 
a "O" is written into storage location me30, indicating that there is no relative image motion at the location of missing 
pixel x30. 

In addition to computing a motion detection map for even field E2, a motion detection map is computed for odd field 
25 01 , again on the basis of thresholding a block of N x N pixels. As in the previous description for the even field the N x 
N block may comprise a 3 x 3 block of pixels. 

Thus, in the present example of a three-by-three summation array, for the above examples of missing pixel locations 
in the odd field Ol , namely missing pixel locations x9, x21 , x1 , x2, x26 and x30, their associated sum signal values may 
be represented as follows: 

30 

Sum x9 = (1 /9)*(|d2 - b2| + |d3 - b3| + |d4 - b4| + |d8 - b8| + |d9 - b9| + |d10 - b10| + |d14 - b14| + (10) 

|d15-b15| + |d16-b16|). 

35 Sum x21 = (1/9)*(|d14-b14| + |d15-b15| + |d16-b16| + |d20-b20| + |d21 - b21| + |d22 -b22| + (11) 

|d26 - b26| + |d27 - b27| + |d28 - b28| ). 
Sum x1 = (1/4)*(|d1-b1| + |d2-b2| + |d7-b7| + |d8-b8|). (12) 

40 

Sum x2 = (1/6)*(|d1-b1| + |d2-b2| + |d3-b3| + |d7-b7| + |d8-b8|) + |d9-b9|). (13) 

Sum x26 = (1/6)*(|d19-b19| + |d20-b20| + |d21-b21| + |d25-b25| + |d26-b26|) + |d27-b27|). (14) 

45 Sum = (1/4)*(|d23-b23| + |d24-b24| + |d29-b29| + |d30-b30|). (15) 

As described above for the even field, each sum signal value Sum xj is compared with threshold T. If the sum signal 
exceeds the threshold value T, then it is concluded that there is effective relative image motion between the first and 
second image fields, and a "1 " is written into storage location (mi=moi ) of the odd field storage space represented by 
so the array M in Figure 9. Otherwise, a "O" is written into storage location (mi=moi), indicating that there is no relative 
image motion at the location of missing pixel xi of the odd fields 02. 

Namely, moi = 1 , if Sum xj > T (16) 

55 0, otherwise. 
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As pointed out above, once the two intermediate motion detection arrays for the odd and even field pairs have been 
derived, a final motion detection map M is generated by logically ORing, point-by-point, the contents of the two (even 
and odd) motion detection arrays, i.e., each respective point mi in the final motion detection map is defined by the 
expression mi = (mei)V(moi) , where V denotes the logical OR operation. The deinterlacer 62 of Figure 6 then proceeds 
to process the odd field 01 and the even field E2 supplied motion detection system 61 , as described above. 

Although the foregoing method may can be carried out using a respective even and odd field motion detection 
storage space or device having a capacity corresponding to that of each of the fields 01 , E1 , 02, and E2, it is possible 
to reduce the amount of hardware required to perform the above process by employing a limited sized storage space 
for processing pixel data to determine the final motion detection map. 
10 More particularly, what is required is a fourth or auxiliary storage space having a storage capacity of N lines of pixels, 
where N is less than the number of lines of the image field. This auxiliary storage space is shown as buffer B in Figure 
9. The manner in which successive lines of the even field E2 can be stored in N-line buffer B and used to generate the 
values of an even field motion detection map is diagrammatically shown in Figures 1 0(a) - 1 0(c) and proceeds as follows. 
In the course of processing the successive image fields ((E1-01-E2-02) of the four field sequence of the present 
15 example, the respective pixel data values ai of the first (even) field E1 are written into storage locations qi of even field 
store FS1 , and then the respective pixel data values bi of the second (odd) field 01 are written into storage locations ri 
of odd field store FS2. For the next or third field (even field E2) in the four field sequence, respective pixel values ci are 
sequentially written into the N-line buffer B, in the order c1 , c2, c3, .... As the third field pixels is loaded into the auxiliary 
buffer B, buffer B will eventually contain a sufficient number of pixel values to permit a binary motion value to be computed 
20 from its contents and those of the first image store FS1 . 

More specifically, at the moment pixel value c8 of the second even field E2 is written into auxiliary buffer B, a binary 
value me1 can be computed for missing pixel location x1 , and stored in motion detection memory M, as described above 
with reference to equation (6). When the next even field pixel value c9 is written into auxiliary buffer B, the second binary 
value me2 can be computed for missing pixel location x2 as described above with reference to equation (7), and so on, 
25 with successive binary values of motion detection memory being computed up to and including value me1 2 when buffer 
B is loaded with even pixel value c18, filling the buffer. At this time the contents of buffer B are as shown in Figure 10(a). 

Once the last pixel value d 8 is loaded into the last (third) line of buffer B, further loading of buffer B cannot proceed 
unless accommodation is made for replacing already loaded buffer space with additional pixels of the image field. For 
this purpose, in accordance with the invention, the first line of pixel values c1-c6 of buffer B are transferred into the first 
30 line of even field store FS1 and written over its current values a1 -a6, so that for i=(1 -6), qi=ci . 

With the pixel values d -c6 in the first line of buffer B having been transferred to the first line of even field store FS1 , 
buffer B is now loaded with additional pixel data, in particular, the next six pixel values c19-c24 of the second even field 
E2, as diagrammatically illustrated in Figure 10(b). As these new pixel values are written into buffer B, the binary motion 
detection values me1 3-me1 8 are computed and written into motion detection memory M, using the procedure described 
35 above. 

When the last pixel value c24 is loaded into the first line of buffer B, buffer B is again full. To again provide for replacing 
already loaded buffer space with additional pixels of the image field, the second line of pixel values c7-c12 of buffer B 
are transferred into the second line of even field store FS1 and written over its current values a7-a12, so that for i=(7- 
12). qi=ci . With the pixel values c7-c12 in the second line of buffer B having been transferred to the second line of even 
40 field store FS1 , buffer B is then loaded with additional pixel data, in particular, the next six pixel values c25-c30 of the 
second even field E2, as shown in Figure 10(c). As these new pixel values are written into buffer B, the binary motion 
detection values me1 9-me24 are computed and written into motion detection memory M, using the procedure described 
above. 

When the last pixel value c30 is loaded into the second line of buffer B, buffer B is once again full. At this time, the 
45 third line of pixel values d 3-c1 8 of buffer B are transferred into the third line of even field store FS1 and written over its 
current values a13-a18, so that for i=(13-18), qi=ci . In addition, since the first and second lines of buffer B contain the 
fourth and fifth lines of the even field, these values are now transferred to the fourth and fifth lines of even field store 
FS1. As a result the remaining motion detection binary values me25-me30 can be computed and written into motion 
detection memory M. At this point in the process, even field store FS1 contains the pixels c1-c30 of the second even 
so field E2 (having been transferred from buffer B, as described above), odd field store FS2 contains pixels b1 -b30 of odd 
field 01 , and memory M contains motion array binary values me1-me30. 

Following computation of all thirty motion detection values mel-me30 for the even fields E1 and E2, the process 
initiates a similar process for the odd field and, in addition, uses the already computed even field data to logically combine, 
on the fly, the motion detection values between the even and odd fields. 
55 More particularly, for the final or fourth field (odd field 02) in the four field sequence, respective pixel values di are 
sequentially written into the N-line buffer B, in the order di , d2, d3, .... As in the case of even field processing, described 
above, in the course of loading the fourth field (odd field 02) pixels into the auxiliary buffer B, buffer B will again eventually 
contain a sufficient number of pixel values to permit a binary motion value to be computed from its contents and those 
of the second odd field store FS2. 
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Namely, when pixel value d8 of the second odd field 02 is written into auxiliary buffer B, a binary value mo1 can be 
computed. Once this value is computed, it is logically ORed with the value of me1 previously stored in motion detection 
memory M and written in place of the value of me1 currently stored as motion detection value ml in memory M. When 
the next pixel value d9 of odd field 02 is written into auxiliary buffer B, the second binary value mo2 is computed as 
5 described above with reference to equation (13). This value of mo2 is then logically ORed with the value of me2 previously 
stored in motion detection memory M and written in place of the value of this currently stored value as motion detection 
value m2, and so on, with successive binary values of motion detection memory being computed up to and including 
value m12, when buffer B is loaded with even pixel value d18, filling the buffer. At this time the contents of buffer B are 
as shown in Figure 1 1(a). 

10 As in the case of processing the even field E2, described above, once the last pixel value d1 8 of the third line of the 
odd field 02 has been loaded into the last (third) line of buffer B, further loading of buffer B involves replacing already 
loaded buffer space with additional pixels of the odd image field 02. Therefore, as described above, the first line of pixel 
values d1-d6 of buffer B is replaced with additional pixel data, in particular, the next six pixel values d19-d24 of the 
second odd field 02, as diagrammatically illustrated in Figure 1 1 (b). As these new pixel values are written into buffer B, 

15 the binary motion detection values mo13-mo18 are computed and logically ORed with the even field motion detection 
values me13-me18 currently stored in memory M. The resulting logical sums are then written into motion detection 
memory M, replacing the values me13-me18. 

When the last pixel value d24 of the fourth line of the odd field 02 is loaded into the first line of buffer B, buffer B is 
again full. The buffer B is now loaded with additional pixel data, in particular, the next six pixel values c25-c30 of the 

20 second odd field 02, as shown in Figure 1 1(c). As these new pixel values are written into buffer B, the binary motion 
detection values mo19-mo24 are computed, logically ORed with the respective motion detection values me19-me24, 
and the logical sums are written over the motion detection values me19-me24 in motion detection memory M. 

When the last pixel value c30 is loaded into the second line of buffer B, buffer B is once again full. Since the first 
and second lines of buffer B contain the fourth and fifth lines of the odd field 02, the remaining motion detection binary 

25 values mo25-mo30 can be computed, logically ORed with the respective motion detection values me25-me30, and the 
logical sums written over the motion detection values me25-me30 in motion detection memory M. 

At this point in the process, even field store FS1 contains the pixels d -c30 of the second even field E2 (transferred 
from buffer B, as described above), odd field store FS2 contains pixels b1-b30 of odd field 01, and motion detection 
memory M contains the logical sums of the even and odd motion array binary values as values m1-m30. As a conse- 

30 quence, the deinterlacer 62 has all the necessary information to generate a deinterlaced frame. 

In the case of fields of color video signals, the memory efficient motion detection system described above is applied 
to the luminance component of the signal. Consequently, if the input data is not represented by a luminance and two 
associated chrominance channels (YCC), it is necessary to perform a transformation to determine the luminance com- 
ponent (e.g. a RGB-to-YUV transform). In the deinterlace system of Figure 6, in the preferred embodiment, merging and 

35 spatial interpolation is applied to each of the three color channels individually, according to the motion detection map. 

Advantageously, currently available digital signal processing circuitry is fast enough to perform motion detection for 
a missing pixel location (x,y) before the next pixel (x+1,y) is acquired. A typical NTSC video digitizer digitizes a video 
signal to 640 pixels. NTSC signals comprising 525 Iines/(iy30) sec. provide 15.75 lines/msec. For 640 pixels, the pixel 
time is 10.08 microseconds, which is well within the cycle time of present day processors. For example, the clock cycle 

40 for a 1 4 MHz processor is 720 nanoseconds. The arithmetic operations to be performed for motion detection within this 
time interval include, at most, nine subtractions, nine comparisons (for absolute values), eight additions and a single 
multiplication (to form a normalized Sum). By "at most" is meant that the number of operations decreases for pixels at 
boundaries with decreasing window size. Also, the summation value Sum can be efficiently computed for two neighboring 
locations; as a three-by-three measurement window slides through the pixels, it is not necessary to recompute the 

45 differences over all nine pixels. Also, within a computation cycle for the first motion detection array me, data may be 
transferred from one frame store to another. 

As will be appreciated from the foregoing description, the hybrid deinterlace scheme of the present invention provides 
an effective mechanism for selecting, on a pixel-by-pixel basis, between merging and (edge-adaptive) spatial interpola- 
tion, in accordance with relative image motion at locations of the missing pixels of a deinterlaced image frame, so that 

so the reduced complexity advantages of using merging in steady regions of the image may be maintained, yet obviates 
the problem of edge-based artifacts, by employing an edge-adaptive spatial interpolation method when relative motion 
is detected. In order to enhance the accuracy of motion detection two consecutive fields of the same polarity are utilized 
for motion detection. The supposed hardware penalty of using such a multifield comparison approach is circumvented 
by employing one of the fields for motion detection, only, so that a reduced size buffer may be used tor the purpose. 

55 

PARTS LIST 

61 Memory Efficient 4-Reld Motion Detection System 

62 Deinterlacer 
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FS1 Even Field Store 

FS2 Odd Field Store 

M Motion Detection Memory 

B Auxiliary Buffer 

Claims 

1 . A method of generating a digital output image comprised of successive lines of pixels from a plurality of immediately 
sequential first, second, third and fourth interlaced image fields having successively opposite field polarities, said 
plurality of interlaced image fields comprising first and third image fields of a first field polarity and second and fourth 
image fields of a second field polarity, interposed with said first and third image fields, said method comprising the 
steps of: 

(a) for a respective one of the successive lines of image pixels of said third image field, generating a respective 
line of output image pixels of said digital output image as corresponding to said respective one of the successive 
lines of image pixels of said third image field; and 

(b) for a respective pixel location of each line of said digital output image for which there is no line of image 
pixels in said third image field, selectively generating a missing image pixel as either a pixel located at the same 
pixel location in said second image field, or generating said missing image pixel as a function of one or more 
pixels in said third image field, in dependence whether a prescribed combination of said first, second, third and 
fourth image fields reveals motion at said respective pixel location. 

2. A method according to claim 1, wherein step (b) comprises selectively generating said missing image pixel in said 
third image field as a pixel located at the same pixel location in said second image field in dependence upon a lack 
of image motion at said respective pixel location. 

3. A method according to claim 1 , wherein step (b) comprises generating said missing image pixel as a function of 
one or more pixels in said third image field, in dependence upon the occurrence of image motion at said respective 
pixel location. 

4. A method according to claim 3, wherein step (b) comprises generating said missing image pixel as a function of a 
plurality of pixels in said third image field, in dependence upon the occurrence of image motion at said respective 
pixel location. 

5. A method according to claim 3, wherein step (b) comprises generating said missing image pixel as vertical spatial 
interpolation of pixels in said third image field adjacent to said missing image pixel. 

6. An apparatus for generating a digital output image comprised of successive lines of pixels from a plurality of imme- 
diately sequential first, second, third and fourth interlaced image fields having successively opposite field polarities, 
said plurality of interlaced image fields comprising first and third image fields of a first field polarity and second and 
fourth image fields of a second field polarity, interposed with said first and third image fields, said apparatus com- 
prising: 

an image motion detection unit which is coupled to receive said first through fourth image fields and is oper- 
ative to selectively output two of its four input lields and a motion detection map, contents of said motion detection 
being representative of whether motion exists at pixels of said digital output image in dependence upon the image 
contents of said first, second, third and fourth image fields; and 

a deinterlacer which is operative to selectively generate missing image pixels of said digital output image in 
accordance with the contents of said second and third image fields based upon the contents of said motion detection 
map. 

7. An apparatus method according to claim 26, wherein said deinterlacer is operative to selectively generating missing 
image pixels of said digital output image as either pixels in said second image field, or as a function of one or more 
pixels in said third image field, in dependence upon the contents of said motion detection map. 

8. An apparatus method according to claim 27, wherein said deinterlacer is operative to generate a respective line of 
output image pixels of said digital output image, for a respective one of the successive lines of image pixels of said 
third image field, corresponding to said respective one of the successive lines of image pixels of said third image 
field and, for a pixel location of each line of said digital output image for which there is no line of image pixels in said 
third image field, to selectively generate a missing image pixel as either a pixel located at the same pixel location 
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in said second image field, or generate said missing image pixel as a function of one or more pixels in said third 
image field, in dependence whether said motion detection map reveals motion at said respective pixel location. 

9. An apparatus method according to claim 26, wherein said deinterlacer is operative to selectively generate said 
missing image pixel in said third image field as a pixel located at the same pixel location in said second image field 
in dependence said motion detection map indicating a lack of image motion at said respective pixel location. 

10. An apparatus method according to claim 26, wherein said deinterlacer is operative to generate said missing image 
pixel as a function of one or more pixels in said third image field, in dependence upon said motion detection map 
indicating the occurrence of image motion at said respective pixel location. 
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(57) A hybrid deinterlace mechanism employs 
motion detection as a precursor to select the value for 
each missing pixel of a deinterlaced frame, which is gen- 
erated by using four consecutive fields to produce a 
motion detection map that is subsequently coupled to a 
deinterlacer, together with a selected two of the four con- 
secutive fields. These two selected fields and the motion 
map are sufficient for the deinterlacer to completely char- 
acterize the resulting deinterlaced frame. The deinter- 
lacer switches between vertical interpolation and 



merging based upon motion. Motion is detected by 
applying a logicOR operation to respective even and odd 
field binary motion detection signals. In the absence of 
motion, the pixel values of an odd field and its immedi- 
ately succeeding even field are merged. On the other 
hand, if motion is detected, vertical interpolation is car- 
ried out for the even field, by a linear combination of 
immediately adjacent pixels, respectively above and 
below the pixel of interest. 
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